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We investigate the metal-insulator transition (MIT) and phase diagram of the half-filled Fermi 
Hubbard model with Rashba-type spin-orbit coupling (SOC) on a square optical lattice. The inter¬ 
play between the atomic interactions and SOC results in distinctive features of the MIT. Significantly, 
in addition to the diverse spin ordered phases, a nonmagnetic insulating phase emerges in a consid¬ 
erably large regime of parameters near the Mott transition. This phase has a finite single-particle 
gap but vanishing magnetization and spin correlation exhibits a power-law scaling, suggesting a 
potential algebraic spin-liquid ground state. These results are confirmed by the non-perturbative 
cluster dynamical mean-field theory. 
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The study of quantum many-body effects and new ex¬ 
otic states of matter are currently amongst the main top¬ 
ics in condensed-matter physics [l|,l2|. During the last few 
years, the successful manipulation of ultracold atoms in 
optical lattices [1-Q and the experimental progress in 
the spin-orbit coupling (SOC) of degenerate atomic gases 
[9|-[l3j have made it possible to explore diverse quantum 
phases I4l-l2l|. More recently, optical lattices combined 
with SOC have attracted enormous interests. It was 
shown that SOC plays prominent roles in many fascinat¬ 
ing phenomena, such as non-Abelian interferometry 221 
and magnetic monopole 23, 2^, topolo gica l phase tran¬ 


sitions 


, non-Abelian localization |28l| . 


or emerging 


elativistic fermions 


When further competing with strong atomic interac¬ 
tions, SOC introduces additional degrees of quantum 
fluctuation, giving rise to remarkable many-body ground 
states. For example, the study of the superfluid to Mott 
insulator transition in the Bose-Hubbard model with syn¬ 
thetic SOC has demonstrated that, Rashba-type SOC 
can induce intriguing magnetism in the deep Mott regime 
33-0, as well as an exotic superfluid pha se with mag¬ 
netic textures near the Mott transition 3l|, l37|. Despite 
this, the essential properties of the metal-insulator tran¬ 
sition (MIT) of interacting fermion systems have been 
less achieved. 


In this Letter, we show that SOC can stabilize a quan¬ 
tum nonmagnetic insulating (NMI) phase in a strongly 
correlated fermion system. Such a system described by 
the spin-orbit (SO) coupled Fermi Hubbard model (see 
Eq. (Ill)) has strong implications for realistic electronic 
materials [13] • Our main results are summarized in 
Fig. m which displays a rich phase diagram. First, the 
SOC tends to destroy the conventional antiferromagnetic 
fluctuations. This results in the distinctive features of 
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FIG. 1: (color online). Phase diagram of the half-filled 
Fermi Hubbard model with Rashba-type SOC obtained by 
the cluster dynamical mean-field theory with a 2 x 2 cluster 
at T = 0.05t. The solid line with dots is the phase bound¬ 
ary of the MIT. The purple-colored regions denote the diverse 
spin ordered phases of a:y-antiferromagnet (a:j/-AFM), spiral 
(the green and red arrows indicate the spins have up or down 
2 -components), stripe, and spin vortex (SV) in the Mott in¬ 
sulating regime. For a > Oc, there exhibits a nonmagnetic 
insulating (NMI) phase in the vicinity of the MIT. 


the MIT with diverse spin ordered phases occurring on 
the side of Mott insulator. Significantly, a NMI phase 
emerges in the vicinity of the Mott transition for a > ac 
(a is the strength of SOC). This phase possesses a fi¬ 
nite single particle energy gap with vanishing magnetic 
orders and the spin correlation function exhibits a power- 
law scaling, suggesting a potential algebraic spin-liquid 
ground state. Recently, enormous attentions have been 
paid to the search for the quantum disordered phase in 
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FIG. 2: (color online), (a) Illustration of the SO coupled 
square lattices, which are mapped onto two sets of sublattices 
for spin up (red) and down (blue) respectively. The central 
shaded box denotes the 2x2 cluster, where the dashed lines 
represent the spin-flipped hoppings, (b) Single-particle energy 
spectra for a = O.Stt. (c) Density of states p{E) for non¬ 
interacting fermions with the strength of SOC ol € [0,7r/2]. 


the interacting fermion systems i3-43|. Our results for¬ 
mulate a promising new route to achieve this intriguing 
quantum state through the SO coupled fermions. 

The model.-T\\B Hamiltonian of a two-component 
Fermi gas subject to an optical square lattice is given 
by 

H = —t 'y ^ 'y ' {cl^TZjjCjcyi -i- h.c.) 

<ij> oa' 

+U ^ -f /r (1) 


where t is the overall tunneling matrix element and Cia 
(c|^) denotes fermionic annihilation (creation) operator 
for a fermion of spin a =t, 4- on the lattice site i. The first 
term describes the nearest-neighboring hoppings with the 
hopping matrices given by TZij = exp[fA • (r^ — rj)], 
where A = {I3ay,aax,0) denotes a non-Abelian gauge 


enerated by the laser-induced spin- 
231. In this Letter we set (3 = —a, 

sB-Sii. 


field which can be 
flipped tunneling [2 

which implies that the SOC is of Rashba type 
In this case, the spin-conserved hopping term is propor¬ 
tional to tcosa, and the spin-flipped term is in propor¬ 
tion to tsina. U is the on-site atomic repulsion and p is 
the chemical potential. The particle number operator is 
fii = hjt + fiix with hia = c\^Cia. 

Method.~We study the physical properties of Hamil¬ 
tonian 0 with the non-perturbative cluster dynamical 
mean-field theory (CDMFT), using Hirsch-Fye Quantum 
Monte Carlo algorithm as the impurity solver [4J, . In 

the presence of SOC, we can map the square lattice onto 


two sets of sublattices for spin up (down) respectively, 
as shown in Fig. [UJa). The 2x2 clusters are embedded 
in a self-consistent medium with the Weiss function of 

the cluster represented by g{iuj) = f ), where 

V 9it 9a J 

ga-a- and gas- are the 4x4 matrix corresponding to spin 
conserved and spin flipped Weiss functions. Due to the 
presence of the spin-flipping term in Eq. o, gn and gi^ 
are generally nonzero. The CDMFT incorporates spa¬ 
tial correlations and has been shown to be successful in 
the study of MIT and magnetic orders (ifil - ld^ . In this 
work, we shall investigate the phase diagram on the half- 
filled square lattice with full range of strength of atomic 
interactions and SOC. 

Before proceeding, we first examine the case with 
a = 0 in Hamiltonian o, which recovers the Hubbard 
model on a conventional square lattice. At half filling, 
the Fermi surface for non-interacting fermions is per¬ 
fectly nested and the antiferromagnetic (AF) fluctuations 
can drive the system into an insulator with infinitesimal 
atomic interaction [i^. In the approach of CDMFT, it 
was demonstrated that without AF fluctuations, the MIT 
between the paramagnetic metal and the paramagnetic 
Mott insulator occurs at Uc/t ~ 6.05 [H^. Here in our 
simulations, we allow magnetic orders to set in and find 
that Uc/t is greatly reduced as shown in Fig. [TJ Further, 
we perform a scaling analysis in Fig. [3Ka), showing that 
the interaction strength Uc/t at zero temperature would 
approach to much smaller values in larger clusters. 

M/r.-Now, we turn to the effects of the SOC on 
the MIT. We concentrate on the basic region given by 
a € [0,7r/2] since the relevant physical results are not 
affected in other regions. First, the single-particle spec¬ 
trum is split into two bands [see Fig. [2][b)], with the 
zero energy Fermi surface possessing a particle and hole 
Fermi-pocket around the center and corner of the Bril- 
louin zone. The corresponding Density of states (DOS) 
for non-interacting fermions is shown in Fig. HKc), where 
the zero energy DOS is suppressed and the bandwidth 
shrinks gradually with increasing a. The suppressed zero 
energy DOS reduce the correlation effects on Fermi sur- 



FIG. 3: (color online), (a) Temperature scaling of the inter¬ 
action strength Uc/t of the MIT for a = 0, with 2x2 and 
4x4 cluster repsecively. (b) Evolution of DOS at different 
interaction strength U/t for a = 1.0. 
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face and hence enhance Uc/t of the MIT, whereas the 
shrinking bandwidth tends to make the Mott insulator 
happen at smaller Uc/t. The two effects compete with 
each other, leading to the drastic changes of the MIT 
boundary in the phase diagram. In Fig. [T]we show that, 
away from a = 0 the value of Uc/t is rapidly increased 
due to the suppression of the conventional AF fluctua¬ 
tion on square lattices. Subsequently, the MIT exhibits 
a nonmonotonic behavior as a function of a. Specially at 
a = 7r/2, the MIT occurs at a finite atomic interaction 
with Uc/t = 4.1. 

In order to feature the MIT in the presence of SOC, 
Fig. [31Jb) plots the evolution of DOS at different atomic 
interactions for a = 1.0. We show that, compared to 
a = 0 case, the zero energy spectral peak in the metal 
phase (red line) is largely suppressed by the SOC. Si¬ 
multaneously, two satellite peaks appear corresponding 
to the Van Hove singularity shown in Fig. [2Kc). Then, 
the zero energy peaks are gradually reduced and a gap 
opens with the increase of atomic interactions. 

Fig. SKa) plots the corresponding single-particle gap A 
and magnetization m as functions of U/t for a = 1.0. The 
insulating phase characterized by a non-zero A is accom¬ 
panied by a finite m simultaneously, indicating a mag¬ 
netic order arises. The specific magnetic phases in Fig. [T] 
can be determined by identifying the spin configurations 
on the cluster, see the Supplementary material for more 
details. Fig. [T]shows that, as a increases the system tran¬ 
sits from ly-antiferromagnet (xy-AFM) to spiral, stripe, 
and spin vortex (SV) phases. Qualitatively, this can be 
understood from an effective spin mod el 15 ill . where the 
induced DM-type super-exchange term [52|, competes 
with the Heisenberg coupling, tending to form diverse 
spin phases. However, the effective spin model works 
only for the deep Mott regime with the atomic kinetic 
energies being treated perturbatively. In close proximity 
to the more interested Mott transition, such a perturba¬ 
tive description breaks down and the strong fluctuations 
arising from SOC may destroy the magnetic orders and 
trigger an order to disorder transition. To address this 
issue, one needs to implement a non-perturbative method 
such as CDMFT to explore in detail the phase diagram 




FIG. 4: (color online). Single-particle gap A and magneti¬ 
zation m as functions of the interaction strength U/t for (a) 
a = 1.0 and (b) a = 1.5, respectively. 



FIG. 5: (color online). Phase diagram in the vicinity of MIT 
for a being close to 7r/2 and T = 0.05t, obtained with 2x2 
and 4x4 cluster respectively. The stars mark the parameters 
employed in Fig. (6] Inset: Temperature dependence of the 
phase diagram for a = ti/2. 


as in Fig. [T] 

NMI phase.-To our surprise, despite the robustness of 
the diverse spin phases in the Mott insulating regime with 
up to modest values of SOC, a NMI phase is found to 
emerge in the vicinity of the MIT for a> Uc (oc — 1-43). 
The NMI phase is characterized in Fig. HKb), where the 
single-particle gap A and magnetization m occur for dif¬ 
ferent atomic interactions C/c^ and Uc^ ■ Specifically in the 
intermediate region C/c^ < D < Uc 2 , the system enters 
into an insulating state but with no long-range magnetic 
order. This is a unique feature of the SO coupled square 
lattice for a being close to 7r/2, where the single-particle 
hopping becomes nearly spin-flipped and the DOS is al¬ 
most suppressed at zero energy. 

The emergence of the NMI phase is further confirmed 
on a 4 X 4 cluster. The larger size of cluster incorpo¬ 
rate more spin correlations and thus, a better description 
of the atomic correlations and SOC induced fluctuations 
can be expected. Fig. [5] plots the phase diagram for a 
being close to 7r/2. We found that, in the 4x4 clus¬ 
ter, the regime of the NMI phase is slightly expanded, 
demonstrating that the NMI is robust in this system. 
We further show, in the inset of Fig. [51 the temperature 
dependence of the NMI phase. The interval between the 
metallic and SV phases enlarge with decreasing tempera¬ 
ture. It demonstrates that the NMI phase is more stable 
at low temperatures by the suppression of thermal fluc¬ 
tuations. 

The NMI phase breaks neither spin nor lattice symme¬ 
try, suggesting a potential spin-liquid (SL) ground state. 
Such a fundamental state was first proposed by Anderson 


54j and has long been sought in the frustrated spin sys¬ 
tems (bij i. Recently, interacting fermion models have at¬ 


tracted wide attentions [4fll - l43l | , and it was reported that 
a SL state can be identified on honeycomb lattice be¬ 
tween semimetal and AF insulator with 3.5t < U < 4.3t 
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FIG. 6: (color online). Staggered spin-spin correlation func¬ 
tion for a = 1.52 and 1.54 with different interaction strength 
U/t marked by the stars in Fig. [5] The dashed lines are 
representative power-law fits to the data. 


[40l| . Despite this, its presence has been challenged since 
the interval of the SL phase is small, which may vanishes 
under the size scaling 5^-5^- The latest results using 
large-scale quantum Monte Carlo (QMC) showed that, 
if the SL state exists, the possible regime reduces sub¬ 
stantially to a small interval 3.8t <U< 3.9t [H^. Sim¬ 
ilar situations have been encountered for the staggered- 
flux model on a square lattice [H, Here, the es¬ 
sential feature characterizing the present system is the 
considerably large space of parameters, where the NMI 
phase emerges. This is in sharp contrast to the limited 
phase space (3.4f < U < 3.9t) obtained in the interact¬ 
ing fermions on honeycomb lattices [^. Specially, the 
predicted NMI phase occurs until a > ac, showing that 
it is a strong held effect of the SOC. 

The absence of magnetic orders in the NMI phase im¬ 
plies strong short-range spin correlation. However, it 
may decay as a power-law or exponentially. To explore 
this issue, we calculate the staggered spin-spin correla¬ 
tion function 


C(r) = (-!)■■ (SgS^ + SgS^ + SgS?). 


( 2 ) 


as shown in Fig. where the spin correlation functions 
are htted to a power-law as C'(r) ~ 1/r^. In NMI phase, 
we hud that the exponent a is less than 2 with a ^ 1.6 
in our simulations. Whereas in the deep Mott insulating 
regime where spin is ordered, a becomes much smaller. 
Therefore, the NMI phase seems to suggest a candidate 
of algebraic SL. Further studies would be implemented 
by QMC calculations in the future. 

The above phenomena of the intriguing MIT and ex¬ 
otic matter states can be investigated in experiments. 
In optical lattices, the Mott insulating phase can be de¬ 
tected by site-resolved imaging of single atoms 
and the spin textures occurring in the Mott pahse can 
be observed via in situ microscopy 67| or through spin- 


resolved time-of-flight measurements 6^. On the other 
hand, the spin correlation can be measured by the spin 
structure factors in optical Bragg scattering [69|, which 
may present the signatures of the spin ordered phases and 
the power-law scaling of the NMI phase. In addition, an 
extremely low-temperature has been recently realized to 
approach the superexchange energy scales [70| . 


In summary, we have investigated the half-filled Fermi 
Hubbard model with Rashba-type SOC on a square lat¬ 
tice. We show that this system displays a rich phase 
diagram. The interplay between the atomic interactions 
and SOC results in distinctive features of the MIT with 
diverse spin ordered phases occurring on the side of Mott 
insulator. Near the Mott transition, a quantum NMI 
phase is found to emerge in a considerably large regime 
of parameters due to the strong field effect of the SOC, 
formulating a new avenue to achieve the intriguing quan¬ 
tum disordered state beyond the spin systems. 
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In this Supplementary Material, we describe how to determine the diverse spin phases and the corresponding 
magnetization. First, in the approach of CDMFT, the Weiss function of the A^-site cluster embedded in a self- 
consistent medium is determined by the cluster self-energy E(za;) via the coarse-grained Dyson equation [l|, 0 


T.I 


K 


iuj -\-— t(K) — 


—1 


+ S(za;), 


( 1 ) 


where t(K) is the Fourier-transformed hopping matrix with wave vector K in the cluster reduced Brillouin zone of 
the superlattice. Note that, in the presence of SOC both the above Weiss function and the self-energy of the cluster 

and column fermionic field operators: 'L'l' = and and define the cluster Green’s function as 


in Eq. ([T|) become g{iuj) = ( 
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. In this case, we introduce two-component row 
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Once g{iuj) is determined, the impurity solver can be used to compute the cluster Green’s function G{iuj). Eventually, 
by using the Dyson equation S(iw) = g~^{iuj) — G~^{iuj), the self-consistent iterative G{iuj) is obtained. 

The energy gap A can be derived by the local density of states (LDOS). By implementing the analytic extension 
of the imaginary time cluster Green’s function G(iui) via the maximum entropy method Q, we have 

p{uj) = 'Y. w) ~ -ilm[Gii(a;)]. (3) 
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Then in the spectrum of LDOS, we can obtain the energy gap A by the energy width of zero density of states. 

The different spin phases in the Mott insulating regime can be characterized by the spin structure factor S'q = 
I with q the 2D wave vector. Here, = (S/ denotes local magnetic order parameter on site i of the 

cluster, with three components given by 

= iRe[Gat(0+) + Gat(0+)], (4) 

>5/ = = -ilm[G,,t.^(0+) - G,,^t(0^)]: (5) 

SI = = iRe[G..n(0+) - Gat(0+)]. (6) 

The spin ordered phases in Fig. 1 of the main text are derived on 2 x 2 cluster, where the structure factor of the 
xy-AFM has a peak at q = (7r,7r), the stripe phase at q = (0,7r), and the SV phase at q = (tt, 0) and q = (0, tt). 
Between the a;y-AFM and stripe phases, spiral phases where the spins spiral in the z-q plane with q = [q, tt) the 
in-plane wave vector may appear. However, the spiral phase is hard to be explicitly identified on 2 x 2 cluster. To 
overcome this difficulty, we explore on a larger 4x4 cluster, and a spiral-4 phase with spatial period of 4 x 2 lattice 
sites is clearly identified in the following FIG. I. 

Finally, we present the definition of magnetization. In the spin ordered phases, we can rotate the local magnetic 
order parameter on each cluster site to a global coordinate system; S' = {7with cfn the angle between the local 





2 


and global coordinates. Then we can define m = ^ with the magnetization given by = Yl,a=x y z ^a- ^or 

example, in the xy-AFM shown in Fig. 1, there are two sublattices {(pi^A = 0, = t^), we have m = A 

where Ci = ±1 is for sites belonging to sublattice A{B) respectively. This general definition of magnetization is also 

applied to other spin phases throughout this paper. 
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FIG. 1: (color online). Spin phase diagram in the Mott insulating regime with U/t = 7.5 on the 4x4 cluster. The intervals of 
the xy-AFM, SV and stripe phases agree well with those on 2 x 2 cluster. Specihcally, a spiral-4 phase with spatial period of 
4x2 lattice sites (the green and red arrows indicate the spins have up or down ^-components) is explicitly identified between 
the xy-AFM and stripe phases. The shaded area indicate other commensurate or non-commensurate spiral phases. 
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